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Abstract 

Software is a communication system. The usual topic of commu- 
nication is program behavior, as encoded by programs. Domain- 
specific libraries are codebooks, domain-specific languages are 
coding schemes, and so forth. To turn metaphor into method, 
we adapt tools from information theory — the study of efficient 
communication — to probe the efficiency with which languages and 
libraries let us communicate programs. In previous work we devel- 
oped an information-theoretic analysis of software reuse in prob- 
lem domains. This new paper uses information theory to analyze 
tradeoffs in the design of components, generators, and metalan- 
guages. We seek answers to two questions: (1) How can we judge 
whether a component is over- or under-generalized? Drawing on 
minimum description length principles, we propose that the best 
component yields the most succinct representation of the use cases. 
(2) If we view a programming language as an assemblage of met- 
alanguages, each providing a complementary style of abstraction, 
how can these metalanguages aid or hinder us in efficiently de- 
scribing software? We describe a complex triangle of interactions 
between the power of an abstraction mechanism, the amount of 
reuse it enables, and the cognitive difficulty of its use. 

Categories and Subject Descriptors D.2.I3 [Software Engineer- 
ing]: Reusable Software; D.2.10 [Software Engineering]: Design 

General Terms Design, Theory 

1. Introduction 

"Metaphor is an invitation to see the world anew.... Metaphor 
transfers meaning from one domain into another and thereby en- 
riches and enhances both domains. " pj 

The design theorist Donald Schon wrote extensively on the role 
of metaphor in design. One of his most famous ideas is that of 
generative metaphor |I7|, which describes a frame or perspec- 
tive carried over from one domain to another to produce new in- 
sights. A consciously embraced metaphor can be enabling, but an 
unacknowledged (tacit) metaphor can cast a 'spell' over problem 
solvers, restricting their ability to see problems objectively. 
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Consciously or not, in software engineering we inevitably find 
ourselves invested in generative metaphors. Particularly prominent 
is Mcllroy's fecund metaphor of "mass-produced software compo- 
nents," which spawned multiple generations of research on soft- 
ware factories, software product lines, software assembly lines, 
software robots, and so forth. 

In this paper we explore the three generative metaphors for 
software components: 

1. Software as communication; 

2. Component design as statistical model-fitting; 

3. Software abstractions as computable functions. 

Software as communication. A fruitful viewpoint for understand- 
ing the role of abstractions in design is that of software as a com- 
munication system. The subject of the communication is program 
behaviors, as encoded in a programming language. Communica- 
tion systems are a primary object of study in the field of informa- 
tion theory, and so information theory can rightly be expected to 
have much to say about abstractions and their role in design. 

Efficient communication can be achieved by identifying frequently- 
occurring patterns or motifs in messages. Messages can then be 
compressed on average by assigning shorter codes to motifs. For 
instance, in spoken English the term "automobile carriage" was 
supplanted in the 20th century by "car," a more efficient form that 
reflects its increased use frequency. These ideas carry over in a 
straightforward way to software. Good software abstractions cap- 
ture commonly occurring motifs, and we can represent our pro- 
grams more concisely (i.e., compress them) by referring to pre- 
defined abstractions, rather than describing them anew for each 
program. Abstractions also compress the design process itself, al- 
lowing us to discuss and reason about designs in terms of recogniz- 
able, high-level chunks. 

The suggested correspondence between software design and 
information theory is summarized by the following table: 



Software Design 

Problem domain 
Program 

Programming language 
Library 
Abstraction 
Code reuse 



Information theory 

Random process 
Message 

Encoding scheme 

Codebook 

Motif 

Compression 



In previous work we developed an information-theoretic view 
of software libraries 1 22 1 . This paper extends this work by using an 
information-theory perspective to analyze tradeoffs in the design 



of components and domain-specific languages. We investigate two 
questions: 

1 . In designing a library for a problem domain, how can we evalu- 
ate whether a component is undergeneralized, overgeneralized, 
or 'just right'? 

2. How and why should we strike a tradeoff between the power of 
abstraction mechanisms in languages, their ease of use, and the 
amount by which they allow program length to be reduced? 

2. Background 

2.1 Information theory of design 

The development of a design discipline around a problem domain 
can be understood as a process whereby people solving design 
problems in a new domain identify recurring motifs in their suc- 
cessful designs, which they abstract into reusable form. As the do- 
main matures, the most useful abstractions form a design canon 
that represents the core knowledge of the design community. From 
an information-theoretic perspective, the design canon is a code- 
book defined to compress (provide terse representations of) design 
solutions. 

We have proposed that a problem domain be associated with a 
probability distribution on programs, where the distribution reflects 
the likelihood that someone working in the problem domain will 
set out to realize a particular program. Many interesting questions 
about design can be reduced, via information theory, to properties 
of this distribution: the scale of abstraction at which we can work, 
the limits of software reuse, and the rates at which software compo- 
nents can be reused. For instance, information theory dictates that 
the extent to which software reuse can occur is governed by the 
'entropy parameter' H of this distribution. In "low-entropy" prob- 
lem domains (with H near 0), programs are highly similar to one 
another and we can design at a high level of abstraction. For prob- 
lem domains with H near 1, the potential for reuse is low and each 
program requires substantial quantities of new code. We developed 
this viewpoint in the paper f22i . 

Tradeoffs in abstraction mechanisms Programming languages 
provide a variety of abstraction mechanisms that serve to capture 
common patterns in designs. Critical to developing good design 
notations and programming languages is understanding the trade- 
offs between various forms of abstraction in terms of succinctness, 
safety properties, complexity, and so forth. A preliminary investi- 
gation of such tradeoffs from the perspective of computability the- 
ory has identified useful avenues of exploration |23|. In this work 
we develop the understanding of such tradeoffs further, extending 
our analysis to encompass connections with the psychology of pro- 
gramming, and develop means to communicate such results to prac- 
titioners and language designers in a meaningful way. One way to 
summarize tradeoffs is to sketch "tradeoff curves" for forms of ab- 
straction, as commonly used in engineering design. Such curves 
give an intuitive appreciation of the tradeoffs inherent in selecting 
forms of abstraction. 

3. Minimum Description Length Principles 

Good component design must strike a balance between over- and 
under-generalization. In this section we describe how the Minimum 
Description Length (MDL) principle, pioneered for choosing statis- 
tical models, also provides a useful framework for reasoning about 
the best level of generalization for a component. 

3.1 The generality problem 

Generality of software components plays a key role in their 
reusability: A general-purpose component is more likely to be 



reused. This has been known since the dawn of time (in software 
terms): describing in 1952 the first major subroutine library for the 
Cambridge EDSAC, the late David Wheeler wrote: 

It may be desirable to code [the subroutine] in such a man- 
ner that the operation is generalized ... |24| 

If a component is 'not general enough,' we call it undergeneralized. 
The development of polymorphism, generics, metaprogramming, 
and so forth has led to languages in which a reasonable level of 
generality is easy to achieve. The ease with which components can 
be generalized occasionally leads to the problem of overgeneraliza- 
tion. Components pay for their generality (or, more correctly, their 
users pay) by requiring too much work to configure, glue, adapt, 
and so forth. As a slightly facetious example we mention that the 
ultimate generalized subroutine in C-l~l- is something such as 

template<typename Inputs , 

typename Outputs , 

typename Operation> 
void do(Inputs& in, Outputs& out, 

Operation& op) 

{ 

op ( in , out ) ; 

} 

This function can be made to do most anything one might 
want; however, to specialize this function to a specific purpose, 
one has to do as much (or more) work than required by a direct 
implementation. (For functional languages, something of the form 
Xx.Xy.xy is analogous.) 

For a more realistic example, consider matrix multiplication. 
The standard Basic Linear Algebra Subroutines (BLAS) function 
for this is called DGEMM. Rather than being merely a matrix 
multiply, DGEMM is generalized to carry out an operation of the 
form 

C ^ aAB + PC 

and optionally, one may transpose any subset of the matrices. This 
broad functionality is paid for by a somewhat unwieldy interface. 
The C binding is: 

void dgemm(char transa, char transb, int m, 
int n, int k, double alpha, double *a, 
int Ida, double *b, int Idb, double beta, 
double *c, int Idc) ; 

The parallel version (from ScaLAPACK) has even more parame- 
ters: 

void pcgemmCchar transa, char transb, int m, 

int n, int k, double alpha, double *a, 

int ia, int ja, int *desca, double *b, 

int ib, int jb, int *descb, double beta, 

double *c, int ic, int jc, int *descc) ; 

Let me emphasize that this is not gratuitous overdesign; there 
are situations in which this flexibility is needed. However, using 
such interfaces requires great concentration from the uninitiate. 
Similarly unwieldy interfaces were commonplace when C-l~l- tem- 
plates were first introduced, and the fashion was to anticipate every 
possible variation with a template parameter. 

Clearly, we must tradeoff the generality of an abstraction 
against the difficulty of applying it. Too specific, and the abstraction 
has limited applicability. Too general, and it becomes arduous to 
adapt. We call the problem of finding the right amount of generality 
for a component 'the generality problem.' 

We propose a solution for the generality problem based on the 
'Minimum Description Length' principle that has proven so infor- 



double sum( double* x, 


int n) 


{ 




double s = 0.0; 




for (int i=0; i < n; 


++i) 


s += X [ i ] ; 




return s ; 




} 





template <typename T> 

T sum(T* X , int n) 

{ 

T s = 0; 

for (int i=0; i < n; ++i) 

s += X [ i ] ; 
return s ; 

} 



(a) Very specific 



(b) Somewhat general 



template<typename T. typename Iter> 

T sum {Iter x, int n) 

{ 

T s = 0; 

for (; x.hasNextO; ++x) 

s += *x; 
return s ; 

} 



(c) More general 



teniplate<typename T, typename Iter 
typename Op> 

T sum(Iter x, Iter end, Op op, 
T unit) 

{ 

T s = unit ; 

for (; X != end; ++x) 

s = op{s,*x); 
return s ; 

} 



(d) Very general 



Figure 1. Four functions that can be used to sum elements of a numeric array. 



mative in statistics 1 16 1. The resulting insights yield practical meth- 
ods to gauge whether abstractions are over- or under-generalized. 

As a running example, consider summing the elements of a 
data structure containing numbers. The C-l~l- function shown in 
Figure[TJi has limited applicability: it can be applied only to arrays 
of doubles. By generalizing this function, we can increase its reuse 
potential. In languages with generics facilities, of which C-I-+ is 
one, one can 'lift' the function of Figure [T^ to a generic algorithm 
1151 15] [TOl 1141 . As a first step one might abstract over the type 
of the array (Figure [TJi). To generalize further, one can abstract 
over the data structure, replacing the array with an iterator | FSl llSI , 
as in Figure [TJ;. One might further generalize over the operation, 
allowing not just summation of elements but also multiplication 
and so forth, shown in Figure[T|l. 

Which of the versions of Figure[T|is the right one? Or, perhaps, 
can this question even be asked in a form that is well-posed and 
suggests an answer? 

3.2 MDL Principles 

There is a strong resemblance between the generality problem for 
software components, and the problem of fitting statistical models 
to data. 

A central problem in statistics is understanding data by fitting 
a model to it. Typically one has a particular model class in mind, 
for example, polynomials. When the models have many degrees 
of freedom, the model can fit the data too closely — for instance, 
finding a high degree polynomial that passes through every data 
point exactly. Such models fail to capture the underlying character 
of the data. 

A solid theory of such tradeoffs has been developed by Rissa- 
nen (161, and separately by Wallace and his colleagues. We follow 
the formulation of Rissanen. The Minimum Description Length 
(MDL) principle states that the model providing the best explana- 
tion of the data is the one providing the shortest explanation of the 
data. This leads to practical methods for model fitting that balance 
the parsimony of the explanation against the closeness of fit to the 
observed data. The MDL principle chooses the model that lets one 
best compress the data; this implies one picks the model most adept 
at finding regularities in the data. 

The MDL Principle is as follows: 



The best explanation of the data is the one that minimizes 
the sum of 

1. the number of bits required to describe the model; and 

2. the number of bits required to encode the data relative to 
the model. 

For example, to encode data with a linear regression model, one 
would first encode the slope and offset of the line, and then encode 
the deviations of the data points from that line. If this encoding 
were more succinct than, say, an encoding using a quadratic model, 
the linear model would be considered a better fit. 

A similar principle can clarify the tradeoff in software design 
between over- and under-generalization. The proposed correspon- 
dence is as follows: 



Statistics 

Model parameters 

Model 

Model class 

Data point 

Underfit 

Overfit 



Software design 

Parameters and glue code 
Abstraction 

Abstraction mechanism 
Use scenario 
Under-generalization 
Over-generalization 



We paraphrase the principle for software components: 

The best level of generality of a software component is that 
which minimizes the sum of: 

1. the code length required to implement the component; 
and 

2. the code length required to adapt the component to the 
desired use cases. 

In the next sections we describe how this can work in practice, and 
what the implications of adopting this principle are. 

3.3 Applying the MDL Principle for components 

The MDL principle requires modification to yield sensible results 
for software components. Our starting point is a set of use cases 
Ui, . . . ,Un and a set of candidate components Ci, . . . ,Ck- We 
assume the abstractions are ordered from least to most general, in 
the sense that the functionality of fi+i is a superset of fi. 



To apply the MDL principle we need an appropriate measure of 
code length. Using bits, as in Rissanen's formulation for statistics, 
would select the component that best compressed the use cases. 
However, this would favour cryptically terse implementations over 
more readable ones. What we need is a code length measure that 
moderates the notion of succinctness with a nod toward usability. 
In preliminary experiments, we have found that the token count 
provides reasonable results. The token count is invariant under 
symbol renamings, comments, whitespace, and so forth, so that 
one cannot make a component 'better' (with respect to the MDL 
principle) by stripping comments and choosing one-letter variable 
names. 

We have found that the following approach yields sensible re- 
sults: 

1 . For each component and use case combination, write code that 
uses the component to implement the use case. If the component 
cannot be adapted to the use case, then write the simplest 
possible implementation of the use case without the component. 

2. For each component, count the tokens required to: 

(a) implement the component; and 

(b) adapt it to each use case. 

3. The MDL principle, as adapted for components, suggests that 
the component minimizing the count of (2) possesses the 'right' 
level of generality. 

3.4 Example 

We illustrate the application of the MDL principle by considering 
three use cases for the candidate components of Figure[T| 

1. Summing an array of double-precision floating point numbers 
(doubles); 

2. Summing an array of integers; 

3. Summing an array of floats. 

In this scenario many experienced programmers would opt for 
the component of Figure [T|b) that abstracts over the data type. 
Abstracting over the data structure or operation provides no benefit 
for these use scenarios, although might be appropriate depending 
on anticipated future needs. 

To determine what the MDL principle suggests, we imple- 
mented four versions of these use scenarios, one for each com- 
ponent in Figure [T| For Figure [TJ a) we merely duplicated the code 
three times, and edited to change the datatypes. For the remaining 
components we 'adapted' them to each use scenario by providing 
appropriate template parameters and arguments. 

Figure|2]shows the code for component Figure[T|d) and the code 
needed to adapt it to the three use scenarios. Using an automated 
tool to count the number of tokens, we obtain the following results: 



Component 

(a) 

(b) 

(c) 

(d) 



Component Adaption Total 



Tokens 

41 

46 

42 

56 



Tokens 

82 

60 

66 

121 



123 
106 
108 
177 



These results are plotted in Figure[3] From the math one expects 
a convex (U-shaped) function, and this can be seen in Figure [3] If 
the component is too specific, it cannot be used for all the use cases. 
If the component is too general, then it is arduous to adapt. And, in 
between the two extremes, we expect a component with the 'right' 
level of generality, which the MDL principle recommends to us. 



The componenl of Figure 2(d) */ 
template<typename T, typename Iter , typename Op> 
T sum(Itei" x, Iter end, Op op, T unit) 
{ 

T s = unit ; 

for (; X != end; ++x) 

s = op{s,*x); 
return s ; 

} 

Code to adapt to use cases 

A function object * / 
template <typename T> 
struct plus { 

T operator()(T x, T y) { return x + y; } 

}; 

/* Three use cases */ 

double sum.double { double * x, int n) 

{ 

return sum(x, x+n , plus <double > () , 0.0); 

} 

int sum.int ( int * x, int n) 

{ 

return sum(x, x+n, plus<int>(), 0); 

} 

float sum_float ( float * x, int n) 

{ 

return sum(x, x+n, plus <f loat > {) , O.Of); 

} 

Figure 2. The component of Figure^d) with the code required to 
adapt it to three use cases. 




□ Abstraction 
■ Adaptation 



Figure 3. Applying the MDL principle for generics. The four bars 
represent the code size required to use the functions of Figure [T] 
for three use cases. Code size is measured in tokens. The light 
bar indicates the size of the generic function, and the dark bar 
shows the amount of code required to adapt the generic function to 
three use scenarios. In (a), there is no generic function and instead 
Figure [T|a) is duplicated for each use case, with a different type 
replacing double. 



3.5 Discussion 

The proposed MDL principle for components can be summarized 
as: 'the best component yields the most succinct representation of 
the use cases/ 

We propose this not as an absolute, but rather as a guiding 
principle. Among the advantages of following this principle are: 

1. Choosing abstractions according to the MDL principle yields 
succinct programs. If we presume a correlation between pro- 
gram length and development/maintenance costs, this suggests 
following MDL principles in component design would be ben- 
eficial. 

2. The MDL principle weeds out components that are overgener- 
alized. Overgeneralized components typically have numerous 
parameters not needed in ordinary usage. These parameters can 
make the component harder to use for novice users, and the pos- 
sibility of variation may, in practice, translate into a possibility 
of error. 

3. The MDL principle provides a quantitative, non-subjective cri- 
terion for choosing the right level of generality. 

However, in practice it would be an unreasonable amount of 
work to approach the design of every component by applying the 
strategy described in Section [53] i.e., implement all the compo- 
nents and use cases and evaluating the description length. We do 
not advocate the MDL principle as a day-to-day design tool, but 
rather for the following uses: 

1. As a teaching tool, a mindset, an exemplar of 'optimal' design 
that can be used to guide more informal decision-making. 

2. As a retrospective tool with which to evaluate existing library 
APIs, with the aim of extracting general design principles and 
recommendations. In a forthcoming paper we apply the MDL 
principle to evaluating the API of some existing generic li- 
braries, notably the STL, with use cases gleaned from open 
source projects. Our preliminary results suggest the STL is 
overgeneralized with respect to the use cases for which it is 
most commonly employed. 

Finally, we note that the MDL principle suggests that the 'right' 
level of generality can only be gauged with respect to a set of use 
cases. These use cases might be chosen to guide the design of a 
component for a specific project, or they may be chosen to represent 
typical usage in a problem domain. In the latter case we can think 
of the use cases as sampling the distribution of programs associated 
with the problem domain. 

4. Abstraction mechanisms 

Modem programming languages support multiple forms of abstrac- 
tion, each with a distinct sublanguage in which they are defined. For 
example, in the language C++ there are distinct sublanguages for 
defining classes, functions, generic functions, and macros. The evo- 
lution of programming languages can be seen in part as an ongoing 
quest to identify useful forms of abstraction and formalize them as 
language features. 

The difficulty of adapting or instantiating an abstraction for a 
particular use scenario appears to relate directly to the computa- 
tional complexity of the problem of inverting the abstraction. The 
cognitive difficulty of spotting common motifs in programs that 
can be abstracted away appears to be closely related to the compu- 
tational complexity of compressing the program with respect to a 
class of abstractions. 

In this section we explore the tradeoff between the power of 
abstraction mechanisms, the degree of succinctness they offer, and 
the cognitive difficulty of their use. 



4.1 Metalanguages as classes of functions 

To examine the differences between forms of abstraction, we need 
a common framework in which to compare them. We believe a use- 
ful viewpoint is that of abstraction mechanisms as classes of func- 
tions, in particular, as classes of partial computable functions. The 
rationale is as follows. An abstraction represents a set of concrete 
instances. For example, a generic linked list class List{T) can be 
instantiated to instances such as List(int) and List{string); we can 
associate with List{T) a function mapping the parameter T to in- 
stances. Similarly, a macro can be associated with a map that sub- 
stitutes parameters into the macro definition; a subroutine can be 
associated with an abstraction function that substitutes arguments 
for variables and inlines the function body; a class definition can be 
associated with a function that imbues subclasses with its function- 
ality; a parser generator can be associated with a map from gram- 
mar specifications to parser implementations. 

This gives abstraction mechanisms an operational interpreta- 
tion, e.g., the activity a compiler would carry out to reduce the ab- 
straction to a lower-level representation. Alternately, we can think 
of abstraction mechanisms as defined by a denotational meaning, 
e.g., we associate with each abstraction a function from parameters 
to object language semantics. In either case, we can associate with 
each abstraction some abstraction function that gives it meaning, 
either operationally or denotationally. 

An abstraction mechanism can then be viewed as a class of ab- 
straction functions, as enumerated by some restricted language we 
call a metalanguage, following the usual terminology of metapro- 
gramming. Programming languages can be regarded as an assem- 
blage of metalanguages, each offering a distinct form of abstrac- 
tioni] 

To fit metalanguages into the framework of computability, com- 
plexity theory, and subrecursive languages, we use the following 
correspondences : 



Software design 

Abstraction 

Abstraction mechanism 

Parameters/glue code 
Instantiation 

Instance of an abstraction 
Adaptation 



Theory idea 

Partial computable 
function 

Class of p.c. functions, 
a metalanguage 
Input 

Evaluation of p.c. function 
Output of p.c. function 
Inversion of p.c. function 



4.2 Facets of abstraction mechanisms 

The fact that programming languages provide a variety of abstrac- 
tion mechanisms (i.e., metalanguages) suggests that there is no sin- 
gle best 'universal abstraction mechanism.' Instead we find that 
metalanguages offer a broad variety of tradeoffs between desirable 
facets, namely: 

• The expressive power of the metalanguage, i.e., what abstrac- 
tions are definable in it. 

• The safety properties we are guaranteed about instances. For 
example, an ongoing concern in programming language design 
is finding metalanguages that can generalize over types in a safe 
way, e.g., generics j9]. 

• Succinctness, that is, how long the description of an abstraction 
must be, and how long parameters must be to produce instances 
of interest. 



' For historical accuracy one might regard a programming language as a 
pastiche of metalanguages. 



double hypot ( double a, double b) 

{ 

return a*a + b*b; 

} 



Figure 4. A simple abstraction that is easy to "invert." 

• The time and space complexity of instantiating an abstraction 
(i.e., how intensive the compilation process must be.) 

• The difficulty of finding parameters to an abstraction that will 
produce a particular instance, i.e., inversion of an abstraction. 

• The effort required to devise an appropriate abstraction, given 
an instance or class of instances over which we wish to gener- 
alize. 

In previous work we used tools from computability theory and 
the theory of subrecursive languages to study tradeoffs between 
succinctness (code length) and safety properties |23|. 

In the present work we examine tradeoffs between the expres- 
sive power of abstractions, the amount of 'compression' they allow, 
and the cognitive effort required to use them. 

4.3 Tradeoffs and cognitive tasks of design 

In designing a compression algorithm, one is is interested in the 
tradeoff between the degree of compression achieved and the com- 
putational cost of compression and decompression. In program- 
ming languages, humans do the compression and compilers do the 
decompression, so to speak. In designing a programming language, 
the tradeoff is largely between the succinctness a language offers 
(i.e., amount of compression) and the cognitive difficulty of recog- 
nizing and exploiting motifs (i.e., the cost of compression). 

Perhaps not surprisingly, the difficulty of cognitive tasks we 
encounter in design appears to correlate with the computational 
complexity of abstraction mechanisms |25|. For instance, a crucial 
design activity is deciding whether an abstraction can be adapted 
to a use scenario. To support rapid design work, the cognitive task 
ought to be simple — abstractions that require great deviousness to 
adapt are unlikely to be used frequently. The equivalent problem 
in our formalism is deciding whether there exist parameters for 
the abstraction function that will cause it to produce a desired 
output — the problem of inverting the abstraction function (not 
to be confused with inverting a runtime computation, an altogether 
different problem.) 

For example, macros and subroutines are forms of abstraction 
"instantiated" by substitution of arguments for variables. Consider 
the function hypot of Figure|4] For the function hypot to be useful, 
it must be possible for us to recognize places in our design where 
it might be used, and to determine what parameters will make it do 
what we want: given the fragment 

double d — c + r*r + f{s) * f{s); 

it is easy to see that the term r * r + f{s) * f{s) can be replaced by 
hypot(r, /(s)). This can be understood as inverting the substitution 
process. The inverse of substitution is unification, which can be 
performed in almost linear time 1131 . It seems significant that 
almost all the abstraction mechanisms we find useful in practice 
lie in low computational complexity classes, and are easy to invert. 
This suggests that we tend to favor simple, easy-to-reuse design 
abstractions over more complicated (but possibly more general) 
ones. Thus, for example, method invocations and inheritance, the 
workhorses of object-oriented programming, both appear easy to 
invert. On the other hand, arbitrary program generation (e.g., as 
in staged languages |19, 20|, generative programming |8|, and 



template metaprogramming I21l l2l[n) tends to be used sparingly 
and often in only simple ways. 

4.4 Viscosity and Lipschitz abstractions 

Cognitive tradeoffs in design notations have been summarized by 
Thomas Green and colleagues in the popular Cognitive Dimensions 
of Notations framework II 11 16|. Green argues that programming 
languages are properly regarded as a medium in which we hash out 
design decisions, not just record them after the fact. Human design 
work — even that of experts — has been shown in numerous stud- 
ies to be disorderly, characterized by false starts, frequent rewriting, 
and simultaneous attacks on the problem at many levels of abstrac- 
tion: "design is redesign, programming is reprogramming." Uli . 
To support the way humans design, notations must be malleable — 
it must be possible to quickly evolve code to match our changing 
understanding of the design. In the cognitive dimensions frame- 
work this quality is dubbed viscosity: the resistance of a notation 
to change. One way to evaluate the 'viscosity' of an abstraction 
mechanism is to analyze how sensitive the input of the abstraction 
function is to small changes in its desired output. Returning to our 
earlier hypot() example, consider a small change in the use sce- 
nario from r * r + /(s) * f(s) to r * r + /(s + 1) * /(s + 1). This 
change that requires only a minor change to the parameters: from 
hypot(r, /(s)) to hypot(r, /(s + 1)). 

This can be formalized by examining the relation between tree 
edit distance [12 26| of the inputs and outputs to the abstraction 
function. Roughly speaking, tree edit distance measures how many 
"editing operations" would be required to transform a term f to a 
term t' , giving a distance metric d{t, t') on terms. If making small 
changes in the instantiated code requires large changes to the pa- 
rameters, we expect the notation to be "viscous" in the sense of 
Green, resisting our efforts to evolve our design. The cognitive di- 
mensions framework suggests that small changes in the instantia- 
tion should be realizable by small changes in parameters. This is 
illustrated in Figure |5] 

We can formalize this intuition in terms of Lipschitz continuity. 
A Lipschitz condition on a real function / : E ^ E is a require- 
ment of the form 

\f{a)-f{h)\<K\a-h\ 

where K > 0. A function satisfying this condition is said to be 
Lipschitz, and K its Lipschitz constant. The notion generalizes 
easily to metric spaces: given a metric space (T, d), and a function 
f : T T,we can call / Lipschitz if 

d{f{a),f{b))<Kd(a,b) 

for some K > 0. Viewing abstractions as functions from terms to 
terms, and tree edit distance as the distance metric, we can make 
the following conjecture: 

The ease with which an abstraction can be used in design 
work is strongly influenced by whether it is Lipschitz, and 
if so, the magnitude of its Lipschitz constant. 

Again, it seems significant that the abstraction mechanisms we 
find useful in practice usually satisfy this requirement. For instance, 
with substitution (the abstraction mechanism for subroutines), the 
edit distance between parameters is at most the edit distance be- 
tween instantiations. This property does not hold in general for ab- 
straction mechanisms that lie in higher computational complexity 
classes, so again we return to the observation that useful abstraction 
mechanisms tend to be computationally very simple. 

4.5 Tradeoff curves 

The formalization of abstraction mechanisms as metalanguages 
lends itself to understanding tradeoffs between forms of abstrac- 
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Figure 5. Illustration of how abstractions interact with edit distance, ideally: a small change in the instantiation (from y to y') can be achieved 
by a small change in the parameters (from x to x'), i.e., d{x, x') < d(y, y'). 



tion: between their generality and ease-of-use, between safety prop- 
erties and succinctness, for example. In engineering it is common 
to sketch tradeoff curves between opposing factors to aid in choos- 
ing the 'sweet spot' where a suitable balance is reached. We can 
draw such curves for abstractions also, and we believe these curves 
provide an intuitive tool for understanding tradeoffs. 
The tradeoff we are interested in here is that between 



Program lengtri 
Effort 

■ Prog, length + effort 



1. the complexity of the abstractions used; 

2. the degree to which programs can be 'compressed' using those 
abstractions; 

3. the cognitive difficulty of using (~ inverting) abstractions. 

Intuitively, if we put 'abstraction complexity' on an x-axis, 
we expect that the program length we can obtain decreases as 
we increase the complexity of our abstractions, and the cognitive 
difficulty increases. 

To draw such tradeoff curves in a meaningful way, we need a 
few justifications; it is not immediately clear, for example, how 
one can put 'abstraction complexity' on the a;-axis in a meaningful 
way. We need a suitable mapping from metalanguages (e.g., classes 
of partial computable functions) to points on an x-axis. If we 
have a chain of metalanguages of increasing complexity, we can 
map this chain onto an axis by appealing to the classical result 
of Cantor that every dense total order without endpoints is order- 
isomorphic to Q, the rational numbers. This suggests we can take 
a set of metalanguages, excise a substructure that is a total order 
(possibly dense), and embed it in the rationals. This provides a 
clean, if somewhat roundabout, explanation for drawing an x-axis 
of metalanguages. This correspondence is not unique, and so the 
placement of particular elements on the axes is arbitrary (up to 
ordering); the observed shape of the graph is meaningful only up to 
squeezing and stretching of the axes. Indeed even the shape of the 
curve itself is usually a sketch based on scanty information, and is 
intended to convey intuition rather than exact information. 

To formalize the notion of 'achievable program length,' we 
could appeal to either Yao-pseudoentropy |3| or a nonuniform 
variant of resource-bounded Kolmogorov complexity. 

To formalize 'cognitive difficulty,' we assume a correspondence 
between the time complexity of the inversion problem and its cog- 
nitive difficulty. 

This is all a little vague, and perhaps the analogy is strained. But 
such graphs can tell useful stories. Figure |6] illustrates the trade- 
off between the complexity of abstractions, their ability to reduce 
program length, and their ease of use. The horizontal axis repre- 
sents abstraction complexity, here normalized so that represents 
no power whatsoever, and 1 represents unrestricted power, e.g., 
Turing-complete program generators. The left vertical axis shows 
the expected program size achievable. As abstraction power in- 
creases, so does the scope for component reuse, and the expected 
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Figure 6. Tradeoffs between abstraction power, component reuse, and 
difficulty of reusing components. 



program size (thick line, left axis) decreases, tending to some opti- 
mal value greater than H(n), the entropy for the problem domain 
1 22 1). It is possible we cannot achieve the maximum possible com- 
pression H{n) because there might be patterns in programs which 
are not exploitable in any effective way, leading to what is labeled 
the "computability gap" in Figure |6] To achieve the best possible 
compression, this curve suggests we ought to use a high level of 
abstraction power, i.e., arbitrary program generators. On the other 
hand, as the power of abstractions increases, the difficulty of using 
them in a given situation (the complexity of the inverse problem) 
increases rapidly, quickly becoming noncomputable (dashed line, 
right axis). Thus we have a tradeoff between the power of abstrac- 
tions to generalize, and the difficulty of adapting them to a particu- 
lar use scenario. The dotted line shows a tradeoff curve with a hy- 
pothetical 'sweet spot' that balances the complexity of abstractions 
against the program length achievable. 

This graph illustrates why in practice we tend to use com- 
putationally weak forms of abstraction, and use complex forms 
of abstraction (e.g., program generators) sparingly, even though 
they might in principle allow us to achieve much shorter program 
lengths. 

5. Conclusions 

We have proposed using the MDL principle to answer the 'gen- 
erality problem,' of how one chooses the right level of generality 



for a software component. As applied to software components, the 
MDL principle suggests that 'the best component yields the most 
succinct representation of the use cases.' In forthcoming work we 
use this approach to retrospectively evaluate the interface design of 
generic libraries. 

The second portion of this paper suggested an approach to un- 
derstanding the tradeoff in programming language design between 
the power of abstraction mechanisms, their ability to reduce pro- 
gram length, and the cognitive difficulty of their use. We observed 
that almost all the abstraction mechanisms popular in practice lie in 
low computational complexity classes. A plausible explanation for 
this is that such mechanisms are easy to 'invert,' e.g., we can readily 
figure out what parameters to provide a macro to achieve a desired 
result. We connected Thomas Green's notion of notational viscos- 
ity to the theoretical notion of Lipschitz continuity, which formal- 
izes the bridge between cognitive difficulty and computational dif- 
ficulty. Finally, we summarized the tradeoffs in abstraction mecha- 
nism design by sketching a curve illustrating the 'sweet spot' that 
balances the complexity of abstractions, their cognitive difficulty, 
and the amount by which they reduce program length. 
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